Autonomous vehicles, such as vehicles which do not require a human driver when operating in an autonomous driving mode, may be used to aid in the transport of passengers or items from one location to another. An important component of an autonomous vehicle is the perception system, which allows the vehicle to perceive and interpret its surroundings using cameras, radar, LIDAR sensors, and other similar devices. The perception system executes numerous decisions while the autonomous vehicle is in motion, such as whether to speed up, slow down, stop, or turn. Autonomous vehicles may also use the cameras, sensors, and global positioning devices to gather and interpret images and sensor data about its surrounding environment, e.g., parked cars, trees, buildings, etc.